Summary

• val_bpb: 0.8495 (3-seed mean, std 0.0013) — updated from initial 0.8508
• Current merged SOTA is Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 at 1.1194
• LeakyReLU(0.9)² activation (based on our 7-point monotonic sweep)
• Backward-looking 5-gram hash cache during sliding window eval (stride=64)
• Fixed-alpha blending (0.8 model / 0.2 cache), no safety gate, no oracle selection
• Verified at stride=2048 (zero overlap): 0.8709 BPB, 97.9% hit rate — not an overlap artifact
• Training ≤ 600s, eval ≤ 371s, all artifacts < 16,000,000 bytes

Results (8×H100 SXM, RunPod)

Current Seeds (v1.1 — sliding window fix + script cleanup)

Training loop exit controlled by MAX_WALLCLOCK_SECONDS=600. Logged wallclock includes torch.cuda.synchronize() overhead (~60-120ms beyond the 600s check).

Overlap Verification

The 0.02 BPB gap between stride=64 and stride=2048 is the overlap contribution. The remaining 0.26 BPB improvement is genuine cache benefit from backward-looking n-gram statistics.

Rule Compliance Checklist

• Artifact ≤ 16,000,000 bytes — All 3 seeds: 15.91–15.92 MB (78–94 KB headroom)
• Training ≤ 10 min on 8×H100 SXM — 600s wallclock, ~6800 steps
• Evaluation ≤ 10 min on 8×H100 SXM — Sliding window eval completes in ~371s
• No training data access during evaluation — Eval paths use val_tokens only
• No training on validation data — Mid-training val checks are inference-only (model.eval() + torch.no_grad())
• N-gram cache is backward-looking — Cache updated AFTER scoring each window
• No oracle/hindsight selection — Fixed alpha (0.2), no min(NLL) comparison, no target-dependent gating
• No external downloads or network calls during eval — Self-contained artifact
• 3 seeds with tight std — std 0.0013 across seeds 42, 1337, 2024
• Cross-model peer review — Independent audit by GPT Codex (gpt-5.4) verified compliance, cache ordering, and artifact sizes against competition rules

Note on N-gram Cache Legality

The competition README does not address n-gram eval caches. No rule in the official documentation prohibits or permits this technique. The README states: "TTT only on tokens already graded" — our cache satisfies this: it is updated only with already-scored tokens. We note that 15+ concurrent PRs (#779, #797, #795, #786, #796, #798, #800, #806, among others) employ the same backward-looking n-gram cache concept.

Architecture

11L, 512d, GQA 8H/4KV, MLP 3×, LeakyReLU(0.9)², XSA (last 4 layers), Value Embedding, BigramHash(2048→128), Partial RoPE(16/64), LN Scale, EMA(0.997), Muon optimizer. Tied embeddings. Mixed int6/int8 quantization + LZMA compression.

Technique: 5-gram Eval Cache

During sliding window evaluation, a hash-based n-gram cache accumulates token statistics from already-scored windows. For each new window, the cache provides empirical next-token probabilities which are blended with the neural model's predictions using a fixed mixing coefficient. The cache is strictly causal — it never sees tokens before they are scored.

This is a pure eval-time technique. No architectural changes, no retraining, no TTT. The trained model is identical with or without the cache.

Related Work

The n-gram eval cache concept has seen significant community adoption since our initial analysis on Issue #140:

• PR Record: 5-gram Eval Cache + LeakyReLU² + Parallel Muon val_bpb: 1.0920 (3-seed mean, std 0.0007) | ~15.9 MB | 8×H100 SXM #659 (@deanbrr) — First n-gram cache submission; ruled invalid for oracle min(NLL) gate, not for the cache concept
• PR Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779 (@deanbrr) — BackoffNgramMixer + Drift-Free TTT (0.6683 BPB)
• PR Record: 11L Full GPTQ + Multi-Order N-gram Backoff (fixed-alpha 0.9757 / entropy-adaptive 0.9605, 3-seed) #778 (@raahilshah) — Multi-order backoff with fixed and entropy-adaptive alpha
• PR Record: 7-gram N-gram Cache (0.8960 bpb) #797 (@armantsaturian) — 7-gram cache (0.8960 BPB)
• PR Record: 11L + order-adaptive 11-gram (mean val_bpb=0.8881) #795 (@hypery11) — Order-adaptive 11-gram (0.8881 BPB)
• PR 0.8128 BPB: Classical Compression Eval + N-gram Backoff on PR #549 Base #786 (@shinegami-2002) — Classical compression + n-gram backoff (0.8128 BPB)
• PR Record: 0.2292 BPB — Dirichlet-Multinomial Smoothing + Distributed Prefill + 15-Gram + EBLS #796 (@Robby955) — Prefill cache + 7-gram entropy-adaptive (0.6567 BPB)
• PR Record: Order-Adaptive Entropy Gating + BackoffNgramMixer (val_bpb=0.5466) #798 (@travispchen) — Order-adaptive entropy gating (0.5466 BPB)
• PR Record: X-WING — Shared N-gram Tables + Cubric (val_bpb=0.5644) #800 (@newjordan) — Shared n-gram tables + Cubric (0.5644 BPB)
• PR Record: Backoff N-gram Cache + LeakyReLU(0.9)² (val_bpb=0.6678) #806 (@ibarrajo) — Backoff n-gram + LeakyReLU(0.9)² (0.6678 BPB)

Our LeakyReLU(0.9)² slope sweep was independently cited by PR #764 (@ndokutovich).

Context

Same team that posted the compliance guide, LeakyReLU slope sweep, and n-gram cache analysis on Issue #140.

Docker: matotezitanka/proteus-pytorch:2.11.0-cuda12.8
RunPod template: Deploy PROTEUS+STYX

Verification

This submission was independently audited by OpenAI Codex CLI (gpt-5.4) as a cross-model peer reviewer — verifying rule compliance, cache ordering, artifact sizes, and training logs against competition rules. Both Claude Code (Anthropic) and Codex (OpenAI) were used throughout development: Claude Code for architecture, implementation, and competition analysis; Codex for independent verification and audit.

Built with PROTEUS+STYX by Light Speed Up